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.1. INTRODUCTION 

- The first implementation of the Leadership, Management 
and Psychology course occurred in the spring of 1970 and 
was documented -in An Analysis and Evaluation of Instructio nal - 
Methodology For A, Multimedia Course iv Leadership, Psycholog y 
arid Management, Phase II Evaluation Report (TR-6.11), and 
Report of Phase II Research Findings; The Design and Methodology 

search on the Interaction of Media, Conditions, of 
Instructipri, and Student Characteristics'for a Multimedi a 
Cburse in Leadership, Psychology and Management, Part I : 
Conditions of Instruction (TR-6.12a). During the summer of 
1970 the materials were revised on the basis of empirical data 
and the course was validated for. a second time in the fall of 
1970. This report describes the fall of 1970 implementation, 
evaluates the effectiveness of the revised materials and 
discusses the procedures used in the final revision cycle 
which preceded the actual installation of the course in the 
spring of 1971. 

During the second implementation, Westinghouse Learning 
Corporation continued its investigation into the relative 
effectiveness of various media and presentation forms 
employed in the course and their relationships to a variety 
of learner characteristics . These results will be presented 
in a Phase III Research Report. 



II. DESCRIPTION OF THE COURSE 

For the benefit of readers unfamiliar with the 
. USNA multimedia Leadership, Psychology and Management 
course, this -section offers a discussion of the developmental 
effort and rationale underlying the course, a description of 
the course as it was constituted in the second validation 
run, and a description of the system used in the fall of 1970 
•implementation. 

■ A. Ndn-Research Aspects of the Course 

Content Outline and Objectives . The content outline 
for the course was developed jointly by WLC and the USNA 
Naval Science Department, Sources for the content outline 
were WLC and USNA subject matter experts, texts on management 
and naval science, and excerpted materials used in the 
traditional USNA Leadership Management course. Reference 
sources were presented in bibliography form at the end of 
the content outline for each part of the course. Once the 
nature of the content was determined, rough drafts of 
behavioral objectives were developed. 

In the process of developing objectives, the actual 
sequencing and outlining of total course content was completed. 
The sequencing of topics within each level of the course was 
generally determined by input from subject matter experts and 
precisely determined by behavioral analysis. See Sequencing 
Rationale . (TP-fi.2) 



Specific behavioral objectives were developed at two 
levels for each topic heading within a segment. Terminal 
objectives (high level problem' solving objectives) were 
developed for each major heading. Enabling objectives 
(lower level objectives) were developed where necessary to 
ensure the learning of information essential to the attain- 
ment of terminal objectives. . ^ 

Content Organizatibn , The content is divided into 
twelve parts, each representing a^ major content area. Each 
■part is divided into varying numbers of segments, ranging 
from as .few as two in Part I to as many as ten segments in 
Part V. A segment is a collection of learning objectives 
closely related by content which can be studied in about an 
hour's time. The segment serves as the logistical [uftit in 
implementation for purposes of scheduling and assessment ofstudent 
progress through the course materials. Appendioc A li?ts the 
titles of each of the parts and segments in the course. 

Depending on -its intended use, a segment is classified 
as core, depth core, or enrichment. The 59 core segments 
include all of the information requisite to the attainment 
of the behavioral objectives. The student is required to 
complete all the core segments, in the order in which they 
are sequenced. Depth core segments are in-class group 
discussions which are structured to provide "in-depth" 
examination of various concepts learned in core segments. 
The selection of depth core segments is made by the instructor. 
In the second validation run, six depth core segments were scheduled 



with required student attendance. Enrichment segments are related 
to but not essential to the mastery o£ terminal objectives. 
They arb optional for students who desire more information 
than that presented in core segments. 

An additional content unit, the module, was created' to 
carry the research experiments which are explained in the 
following section of this report. A module is a vehicle 
employed for the development and presentation of materials in 
which research variables are manipulated. Several parallel 
modules, were prepared in each segment utilized for research 
purposes, representing variations specified by the experimental 
designs. The different modules of a segment are distinguishable, 
from one another by differences in presentation variables 
and/or media, although the content is the same. 

B. Research Aspects of the Course . 

Dimensions of Presentation . Concommitant with the 
identification and sequencing of course content and objectives, 
WLC developed a series of instructional research hypotheses 
which were tested in the first implementation and re tested in 
the fall of 1970 run of the course. A detailed explanation 
of the instructional research design can be found in the 
Report of Phase II Research Findings. Part I: Conditions of 
Instruction (TR-6.12a) and in the forthcoming Phase III * " 

Research Report. In this report only the basic dimensions 
of the research are given. 
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Research hypotheses were based on concepts presented 
A Behavioral Approach to Instructional Design and Med ia 
Selection , in which a distinction is made between the 
importance of media, and presentation forms (Tosti and Ball, 
1969). The distinction is that the medium is only the mode 
of transmission of information whereas the truly important 
characteristics of instruction are the forms of presentation 
of the information within the medium. Any medium can be 
described with respect to its characteristic presentation 
dimensions and wi^h respect to its capability for varying the 
dimensions of presentatioiv. The most basic dimensions of 
presentation which characterize a medium are the form and 
frequency of the stimulus, response, and management decisions. 
In matrix form these dimensions are: 

Form Frequency 

S.ti">ulus Stimulus representation Duration 

. Response . Response demand form Response demand frequency 

Management Management form Management frequency 

Stimulus representation is the form in which the stimulus 
is presented. It can be written, spoken, or pictorial. 
Stimulus duration is the temporal stability of the stimulus. 
Duration can be transient, such as movies and lectures, or 
persistent, such as textbooks or other. printed matter. 

Response demand form, is the type of response the student is 
required to make. The form of the response demanded can be 
covert or .various forms of overt responses . Response demand 



frequency is the frequency with which the student is required 
to respond. The frequency with which questions arc asked 
within an instructional sequence can be high, med5um, or low. 

Management form is the type of remediation the student 
is given. It can be multilev 1 - remediation by differing 
levels of response demari frequency within a single form; 
rultiform - remediation by a different form "of presentation; 
repetition - a simple repeat of the same presentation; or 
error diagnostic - remediation by branching according to 
specific incorrect responses. Management frequency is the 
frequency with which the presentation is repeated or' changed 
according to the need for remediation. The frequency of 
management can be high, medium, or low depending on response 
demand frequency. 

i 
I 

The- Phase II Evaluation Report reviews the rationale 

) ^ ■ 

underlying initial media selection and explains changes in the 
media employed in the scconi imprementation as a result of the 
data from the first run. 

• Presentation Design . The media and presentation forms 
implemented in the fall of 1970 are shown schematically in the 
Presentation Design (Appendix B) . The Presentation Design 
depicts the sequence of segments and the modules (media and 
presentation options) available in each segment. In following 
the Presentation Design, read from left to right first in the 
uppQr schema and then in the lower schema on ei-ch page. The 
student studied the segments in numerical order, seeing only one 
module in each segment. The first box in the upper left corner 



of Appendix B is numbered 1.1, indicating that it is Part 
One, Segment One. The medium in which it is developed, 
Syndactic Text, is shown inside the box (the format of the 
Syridactic Text will be discussed later). Upon completing 
Segment 1.1, the student proceeds to Segment 1.2, which as 
is indicated in the next box is a discussion booklet. 
Segments 1.1 "and 1.2 iare the only segments available in only 
one inedia and presentation form. . The third segment the 
student encounters is Segment 2.1 which is available in four 
different modules (vertically arrayed in four boxes). Above 
each box is a three digit number. The first digit indicates 
the part, the second digit the segment, and the third digit 
is the module identifier. The media and presentation 
differences between the modules are described by the notations 
inside the boxes. The last segment encountered is Segment 12.4. 

Media and Presentation Variables . Media used in the 
fall of 1970 implementation were audiotape/panelbook (AT/PB) , 
AT Script/PB, audiotape/intrinsically programed 4)ooklet (AT/IPB), 
AT Script/IP3, computer assisted instruction (CAI), 
CAI Script/IPB, linear text and syndactic text. A description 
of -the media and each of the variations in presentation form 
(modules) within each medium is given in this section. 

In the audiotape/panelbook (AT/PB) segments (2.2-2.S, 
3.. 1-3.4, 5.7-5.10 and 7.1-7.4), instruction is delivered in 
lecture format by a commercial radio announcer reading from 
a prepared script. The auliotapes were developed in a 
commercial recording facility using standard recording tapes. 



and were transferred to cassette cartridges for student use. 
All charts, photographs, and drawings accompanying the 
audiotape lecture are presented in a panelbook. 

Variations in the modules reflect different combinations 
of three presentation conditions - response demand frequency, 
response demand form and presence or absence of confirmation. 
In addition to illustrations, the panelbook contains questions 
on the instructional material. The frequency with which a 
response is required is either high (about 20 ques tions /panel - 
book) or low (about 3 questions/panelbook) . The form of 
the response the student makes to these questions is either 
covert (he thinks over his answer) or overt (he records his 
selection on an answer sheet) . In some modules confirmation 
to the questions is provided while in others the student 
never receives the correct answer. • . " 

In some segments, (7.5, 9.1 and 9.2) an alternative to 
the audiotape/panelbook is the audiotape script/panelbook 
medium where the printed (as opposed to the audio) version 
of the script is used by the student in conjunction with a 
panelbook. In these segments only one presentation condition, 
presence or absence of content raaps , is manipulated. The 
content map is a hierarchical structuring of the terminal 
and enabling objectives in the segment and is used as an 
advance organizer by the students. 

Another variation in usage of a taped medium is the 
audiotape/intrinsically pr o gramed booklet (AT/IPB) combination. 
The instruction is delivered via the audiotape while questions 



requiring an active response are posed in the intrinsically 
programed booklet. The presentation variables manipulated 
in these segments (4.4-4. 7). are response demand frequency 
and management frequency within the IPB. In the modules 
characterized by high frequency of response (about 20 
questions/IPB), the management frequency ranges from high, 
(error diagnosis on every question) to medium (error 
diagnosis on every other question) to low (no error diagnosis) 
In the low response demand frequency condition (about three 
questions/IPB), management frequency is also low. The 
audiotape/IPB media combination is paralleled in each case 
by another media mix which is hardware- free: the audiotape 
script/intrinsically programed booklet combination. Only 
one IPB version, the high response demand frequency /high 
management frequency version, is used in conjunction with 
the script. 

There are two segment,, ^.l0.1 and 10.2) in which the 
intrinsically programed booklet is used either with an^ 
audiotape or a script and the only presentation dimension 
manipulated is the presence or absence of a content map. 

A different medium, comput-'er assisted" instruction (CAT), 
is used in some segments (12.1 - 12.4) to carry presentation 
variables similar to these tested in the audiotape/IPB 
segments (4.4 - 4»7). The CAI material was developed for the 
1500 Instructional System, utilizing the three components of 
the system - CRT display, audio, and image projector. The 



informational fran,es are presented on the CRT screen and 
image projector. The questions, which are often situations 
in which the student has to choose the best course of action 
are presented: 1) on the audio, where the situation is 
described; 2) on the image projector-, where pictures of ' 
the situation are presented' along with the audio; 3) on the 
CRT screen, where the student is aslced to select an answer. 

In each CAI segment there are four modules in which 
response demand frequency and management frequency are ' 
manipulated j.ust..as they were in Segments 4.4-4.7. I„ 
■ addition, KLC developed a fifth module, the CAI^criEt/ 
IntrinsicallY programed booklet for implementation in the 
fan of ^0 run. The intent of this module is to offer a 
hardware-free version of the CAI material. To more closely 
simulate actual operation of the CAI system, the paper 
version incorporates the script in the IPB (which is the 
high response demand frequency, , high management frequency 
version). 

Printed matter serves as the medium for several types 
of materials prepared by wic. The linear texts (Segments 
4.1-4.3, 5.1-5.6, 8.1-8.3) have befen developed by the RULEG 
and EGRUL methods of programing (Rule-example; example-rule). 
These are essentially programing methods of presenting a 
rule tdefinition, principle) and having the' student identify 
an example of the rule (from 2. 3, or 4 choices), or pre'sent- 
ing an example and having the student identify the rule or 
principle which is depicted in the exampie. 



Variations in the linear text modules are permutations 
of two presentation dimensions - response demand frequency 
and response demand form. In three modules the response- 
demand frequency is high, with a question for each information 
frame. The only difference among these modules is in the 
three forms of response required of the studejit. In one 
form the student is requested to underline the key points 
in the correct response, in another form he records his 
selection (A, Bj C or D) on an answer sheet and in the third 
form he responds covertly - that is, he meraly reflects upon 
his answer. These three forms of response are repeated using 
linear text modules which are characterized by low response 
demand frequency. In these modules a question is asked 
about every tenth frame. The remaining frames which were 
question frames in the high response demand frequency module 
are restructured as information frames. 

The syndactic text (Segments 2.6-2.8, 6.1-6.3, and 
11.1-11.3) contains several summary statements each followed ■ 
by a summary quiz of about five questions. The presentation 
dimensions which are varied among the three syndactic text 
modules are' management frequency, management form and' 
response demand frequency. In two modules there is medium 
management frequency, that is, the student is- sent to the 
next summary statement if he correctly answers all questions 
on the summary quiz or he is required to study other material 
if he incorrectly answers any questions on the summary quiz. 



In the two modules with medium management frequency there 
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I are "two types of management forms: (1) the programed 

sequence, a sequence of linear frames characterized by high 
response demand frequency and (2) the detailed summary, a 
, prose version of the content of the programed sequence, with 
no response demand required. To provide data for materials 
jj revision, a scrambled version of the identical summary quiz 
is readministered after the student completes the programed 
sequence or detailed summary. The third module developed for 
H . syndactic text has no management condition. In this module 

the student simply reads the summary statement and takes and 
I scores the summary quiz. Regardless of his performance on 

■ the summary quiz, he continues to the next summary statement, 
i! In some segments employing syndactic text (2.1, 2.9, 

I 3.5, 6.4) only the module with the linear frames is ■ 

_ used and the variables manipulated are the presence or 
P| absence of a special answer form and content maps. 

Finally in some segments of syndactic text (5.1-5.3, 
'< and 8.4-8.6) only the presence or absence of content maps 
is manipulated. 

1 C; Implementation 

i ■ . . 

, The second implementation of the course in which 

i 

forty-four USNA midshipmen participated was conducted in the 
fall of 1970. The course was administered by a full-time 
staff consisting of an on-site WLC instructor and tv.'O data 
] clerks. The instructor was responsible for tutoring students, 



selecting and guiding group discussions 

(depth core), scheduling and administering USiNA examinations 
and determining grades. The data clerks controlled materials 
distribution, administration of cumulative posttests, scoring 
of progress checks and collection of frame response forms 
and questionnaires. 

Students were routed through the course according to 
procedures outlined in the Student Guide. Each student had 
been preassigned randomly to a particular module in a segment 
and was required to follow exactly his unique track through 
the course siaterials„ The students were allowed to work 
through the materials at their own speed as long as they 
maintained a minimum pace which ensured proper preparation 
for depth core meetings and USNA exams. After studying a 
segment the student took a progress check which was. handed 
out with the instructional materials. (The progress check is 
discussed in detail in a later section of this paper.) The 
progress check was scored by the WLC staff. If the student 
did not attain a score of 801 or better on the progress check, 
he was required to go through a remediation cycle. In* 
remediating, a student reviewed instructional materials 
pertinent to the content missed on the test items, and then 
retook the progress check. If he failed to achieve 801 
cprrect on the second trial of the progress check, the 
student was tutored by the instructor. 

Of the 59 segments in the course, 48 were included in 
groups. of 3 or 4 in research units. Upon" completing each 



research unit the student Was required to take a research 
test, the cumulative posttest (CPT) , which is used" to 
measure the effects of different presentation and media 
conditions on student performance. When the student studied 
a segment in a research unit, he took the progress' check and 
went on to. the next segment. After all the segments in 
the research unit were completed, the student turned in the 
progress checks for those segments, and took the CPT, 
The progress- checks were graded and shown to the student if 
he scored 801 or better. If he failed to achieve the 801 
criterion, he had to remediate. Thus, remediation on^esearch 
segments v^as postponed until after the CPT was taken. In 
the eleven non-research segments, the student's progress 
check was graded and any remediation necessary took place 
before the student began the next segment. 



III. DKSCRIPTION OF EFFHCTIVHNliSS MKASURES 
a1 Test Item Pool 

WLC developed a pool of criterion referenced test 
items to provide the capability of assessing the effectiveness 
of the course. The specification for developing test items 
was thatjtwp test items be developed for each of 12 objectives 
in a segment; there are thus approximately 1400 items in 
the pool (24 X 59 = 1416). The selection of objectives to ' 
be included- was based primarily on the need for 
representative coverage of terminal objectives, and 
secondarily on representative coverage of enabling 
objectives. All test items in the -pool bear a one-to-one 
relationship to behavioral objectives. Since both WLC and 
USNA subject matter experts assisted in the development and 
review of .:he test items, content validity for all items can 
be assumed. Enabling objective's not covered by test items 
in the test item pool ate measured within the course 
materials by criterion frames. 

B. Administrative Pre and Posttest 

Administrative tests were developed to provide an 
instrument for evaluating total course achievement. The 
administrative pre and posttest was actually one 80-item 
test which was administered once at the beginning and 
once, at the end of the course. (The administrative pre 
and posttest is distinct from USNA examinations which were 



administered throughout the course by the USNA instructor 

for purposes of assigning eight-week and final course grades.) 

The administrative pre and posttest was developed by 
representatively selecting test items from the test item 
pool. In this way, there was at least one test item 
selected from each segment of the course, plus an additional 
test item from each of 21 segments. 

It was necessary to delete S test items from the 
analysis of the administrative data because the content for 
these items was revised substantially between the 
administration of the pre and posttests. In addition the 
pre/posttest data for one student was incomplete. The 
pre and postte$t analyses therefore are based on 75 test 
items and 43 students. 

C. Progress Checks 

Progress checks were developed to measure student 
achievement on each of the core segments. They were tests 
made up of a minimum of 10 criterion referenced items drawn 
from the test item pool. On 7 of the 59 segments there were 
more than 10 test items because these segments contained an 
unusually high number of behavioral objectives. . 

As mentioned in the discussion on course implementation, 
the progress checks were handed out with the materials. 
Students were instructed to study the material and then take 
the progress check. The progress checks were -orrected by 
the WLC administrative staff and if the student did not score 
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801 or better, he was required to restudy material pertinent 
to the test items missed and to retake the progress check. 
If after the second trial the student still did not attain 
SOIon the progress check, he was tutored on the questions 
he missed. Thus the instructional system ensured that the 
student would l^^arn the minimum criterion of 801 of all 
progress checks test items. Since there is a one-to-one 
correspondence between the behavioral objectives and the 
criterion referenced tost. items, it can be said that all 
students learned 80* of the measured course objectives. 
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IV. EVALUATION OF COURSE EFFECTIVENESS 

Since the Leadership, Management and Psychology course is 
based on a behavioral approach to instruction, the measurement 
total course effectiveness is based primarily on student 
test performance over a series of behavioral objectives. 
The premise for this form of evaluation is that once 
behavioral objectives are developed for a course and every- - 
one agrees that they are necessary and worthwhile objectives, 
then the test of the effectiveness of the system is simply 
whether students attain the objectives. 

The measurement of. student performance on stated 
behavioral objectives is a technique for assessing the absolute 
effectiveness of a system. Relative effectiveness could be 
assessed by comparing the effectiveness of one system or 
portions of the system to other systems. The relative 
effectiveness of the WLC multi -media system vis a vis the 
effectiveness of the existing USNA Leadership courses has 
not been assessed for the following reasons: 

1. The evaluation of the multi-media course is Jbased 
on -student performance on test items covering over 600 of the 
approximately ISOO behavioral objectives. A comparison of 
the effectiveness of the multi-media course to other courses 
would therefore necessitate the inclusion of over 600 measured 
objectives in the other courses. To compare effectiveness 
based on final examinations alone would mean comparing 
effectiveness on only a small sample of objectives rather 



than on the wider and more reliable sample offered by the 
progress check tests. 

2. A second consideration in multi -media vs. existing 
leadership course comparisons is the possible Hawthorne and 
Rosenthal effects which may bias results. These two effects 
are respectively the tendencies 1) for students to realize 
they are in an experiment and to perforin beyond typical 
expectations (Schramm, 1964) and 2) for teachers to realize 
they are being compared and thus alter their typical patterns 
of instruction (Rosenthal, 1966). 

3. If the effectiveness data for the multimedia course 
and the existing leadership course were not identical, there 
would be no way of accounting for the differences. Since 
several different media and fcrms of presentation are being 
used in the WLC course, and since teaching methods and 
materials vary from one USNA instructor to another, there 
would be no clear cut indication of the conditions of 
instruction which account for total differences. 

4. Within the multi-media course, the effort is made 
to compare the relative effectiveness of one mode of 
presentation to another. In making these comparisons, all 
variables except specially selected presentation or media 
variables are held constant, that is, students are given the 
same content, objectives and test items. Only under these 
circumstances is it possible to state. that one form of 
presentation or media is relatively more effective than 
another. 
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5. The multimedia course is not intended to supplant 
the instructor in other courses. Its intent, rather, is to 
teach effectively the core content requisite to the under- 
standing of Naval leadership, thereby reducing the need for 
the instructor's role as strictly a disseminator of information. 
An instructor using multimedia materials need only augment 
prepared materials with personal guidance of students. That 
is, he is able to select points he would like to highlight, 
lead group discussions, tutor and 'counsel students, and in 
general use his time as a professional to invent new and 
creative ways of simulating leadership experiences. With 
these considerations in mind, studies of the effectiveness 
of systems which essentially compare one role of the 
instructor to another role are too global to be of value. 

A* Evaluation Basod on Administrative Pre/Posttest 

The most direct although somewhat simplistic estimate 
of total course effectiveness is obtained. by comparing 
student performance at the beginning and at the end of the 
course. For this purpose WLC developed a 75- item 
administrative pre/posttest which measures performance on a 
representative sample of the approximately 1500 behavioral 
objectives in the course. This test was given to the students 
before they began. the course and again when they completed it, so 
that a gain from pre to post test could be computed. However 
the actual gain from pre to posttest is a relatively 



Incomplete effectiveness index because it does not take 
into account the amount of total gain that was possible. 

An index which does represent how much students learn 
with fespect to how much they could have learned is the gain 
score ratio. This is the ratio of actual gain to maximum 
possible gain. The formula for the gain score ratio is: 

(Posttest Score) - (Pretest Score) 
If Items on Test) 7 CPretest Score) 

The gain score ratio has been- calculated for each 
student and as can be seen in Table 1, there is a wide 
range in the ratios with a clustering of scores in the 
.^0 to .60 intervals. . 



TABLE 1 

Administrative Tests^ 

Frequency Distribution of Individual Gain 

Score Ratios (N=43) 

Gain Score Ratio Number of S ludents 

.900 to .999 * 0 

.800 to .899 0 

.700 to .799 2 

.600 to .699 7 

.500 to .599 - 12 

.400 to .499 10 

.300 to .399 9 

.200 to .299 2 

.100 to .199 1 

.000 to .099 0 
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Table 2 gives the means of the individual scores used 
in determining the mean gain score ratio of .495 for the 
course. 



. . TABLE 2 

Administrative Tests^ 
Mean Gain Score Ratio 
(t of Test Items=75, If of Students=43) 

Posttest Pretest Mean Maximum Mean Gain 

Mean Score Mean Score Gain Possible Gain Score Ratio 

53.4 32.2 21.2 42.8 .495 



An evaluation of the course based solely on pre/posttest 
performance, therefore, reveals that the students learned 
49.51 of what they could have learned. It should be noted 
that in order to have a mean gain score of 1005, all of the 
students would have to miss all of the questions on the 
pretest and ans\^er them all correctly on the posttest. 

One problem associated with the use of gain score 
ratios in general is that there is no established standard 
by which to gauge them. To indicate that a system is 49.5% 
effective may seem to indicate that the system is not 
operating at an acceptable level of effectiveness. However, 
when the gain score ratio is interpreted in conjunction with 
the posttest average, it becomes more meaningful. In this 
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light, although the students gained only 49.51 of what they 
could have gained, their final scores were fairly 'high, 
averaging 71.2% (53.4/75 x 100). 

To interpret further the gain score ratio it is 
necessary to consider each item of the pre/posttest from 
which the data are derived. Table 3 provides information 
on the percentage of students correctly answering each item 
on the tvo tests. 



TABLE 3 

Administrative Tests^ 
Items Correctly Answered by Percentage of Students 

Number of Items Correctly Answered 

Percentage 

of Students Pretest' CCuml Posttest CCum) 

91.0 toion.0% 
81.0 to 90.9 
71.0 to 80.9 
61.0 to 70.9 . 
51.0 to 60.9 
41.0 to 50.9 
31.0 to 40.9 
21.0 to 30.9 
11.0 to 20.9 
1.0 to 10,9 



0 


0 


18 


18 


3 


3 


14 


32 


7 


10 


8 


40 


10 


20 


14 


54 


9 


29 


7 


61 


10 


39 . 


7 


68 


13 


' 52 


2 


70 


9 


61 


4 


74 


7 


68 


0 


74 


7 


75 


1 


75 



By combining tile last five intervals in Table 3, it can be 
seen that. there are 14 items on the posttest which were 
answered correctly by fewer than half of the students. In 
large part these 14 items account for the low gain score r, 
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It was possible that these items performed poorly " 
because they tested material taught at an early point in the 
course and the students had simply failed to remember the 
material. To investigate this possibility, the 14 test items 
were grouped into 4 quartiles such that quartile 1 represented 
test items on material in the first 15 segments (1.1-3.4), 
quartile 2 the next 15 segments (3.5-5.7), quartile 3 the 
next 15 segments (5.8-8.3) and quartile 4 the last 14 
segments (8.4-12.4). The distribution of the 14 test items is 

Quartile (Segments) Number of Test Items 

1 (1.1-3.4) 2 

2 (3.5-5.7) 5 
• 3 (5.8-8.3) 4 

■ 4 (8.4-12.4) 3 

It would appear, therefore, that the different time spans 
between learning the material and answering questions on it 
was not a factor in the poor performance on these items. 

To investigate furtjier the impact which retention may 
have had on overall performance on the posttest, the 
percentages of correct responses on each posttest item were 
grouped into quartiles according to the segment which the 
test item represented and a mean percent correct was obtained 
for each quartile. 
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TABLE 4 

Posttest Performance by Quartile (by Segment) 

Quartile (Segments) Number of Test Items Mean Percent Correct 

1 .(1.1-3.4) 22 69.9 

2 (3.5-5.7) IS 72.0 

3 C5.8-8.3) 21 68.9 

4 C8.4-12.4) 14 75.4 



Looking at the data in Tables 3 and .4, one may conclude 
that retention did not influence performance on the posttest 
in general or on the 14 specific items which performed poorly. 
Since retention has been eliminated as a possible explanation 
of low scores on some of the posttest items, the poor 
performance probably indicates either that the test item 
itself is faulty or that the material from which it is 
derived needs to be re-examined with respect to the adequacy 
with which it is taught. 

B. Evaluation Based 'on Progress Checks . 

An evaluation of course effectiveness, which is ifiore 
comprehensive than that based on the administrative pre/posttest 
data, is achieved by analyzing progress check data which 
measure performance representatively across each of the 59 
segments in the course. Bearing in mind that there is 
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considerable variation among segments, the "average" 
segment contains 8 terminal objectives and twice as many- 
enabling objectives. A progress check test is taken at 
the end of each segment. The progress checks have a ■ 
minimum of 10 items (7 progress checks have more than 10/ 
ranging up to a maximum of 15 items) . There is a test 
item in each progress check for each of the terminal 
objectives; the remaining items are based on enabling 
objectives. The enabling objectives which are not tested 
on the progress checks are covered by criterion frames 
within the materials. In aggregate the progress "^hecks 
measure performance on 606 of the approximately 1500 
behavioral objectives in the course. Since there is a 
one-to-one correspondence between test items and the 
objectives they measure, performance on progress checks is 
presumed to be a reliable index of performance in the" course 
itself. , 

As described in Section III C, the student who fails 
to answer correctly at least 801 of the progress check test 
items is required to' restudy material tested by the incorrectly 
answered items, then retake the progress check. If he still 
does not obtain a score of 80^ or better on the second iteration, 
the student is tutored by the on-site instructor. Thus, the 
system ensures that the student master at least 80% of the 
measured objectives in the course. ^ 

Table 5 is a frequency distribution of the 606 progroDs 
check items tabulated by the percentage of students answering 
correctly on the first iteration. 



TABLE 5 



Objectives Measured by Progress Checks. 
Frequency Distribution of Objectives 
by Percentage of Students 
Attaining Them - First Iteration 

(# Objectives = 606, # Students = 44) 

I Students Objectives Attained on Progress Checks 
Attaining 

Objectives Number Cumulative ft Cumulative % 

100% 115 115 19.01 

90-99 234 349 57.6 

80 - 89 113 462 76.2 

70 - 79 55 517 85.3 

60 - 69 39 556 91.8 

50 - 59 23 579 95.5 

40 - 49 14 593 97.9 

30-39 6 599 98.8 

20-29 5 604 99.7 

10-19 1 60S 99.8 

0-9 1 606 100.0 



Table 5 reveals that at least 801 of the students correctly 
answered 76.21 of all of the progress check test items on 
the first iteration. This indicates that the materials in 
general are effective instructional tools. By combining the 
last 6 intervals in Table 5, it is seen that there are SO 
(606-556) test items which were correctly answered by fewer 
than 60% of the students on the first iteration. Poor 
performance on these items can be attributed to a faulty test 
Item, inadequate teaching of the objective, or high difficulty 
level of the objective itself. During the revision cycle 
following the fall of 1970, these test items/objectives were 
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examined and many were altered. Improved performance on 
• these items can be expected during the next implementation 
of the course. 

0 The progress check data have been analyzed by segment 

P as well as by individual objective to identify those segments 

which were performing below criterion level. Table 6 is a 
1^ frequency distribution of the segments by the percentage of 
students who correctly answered 80* or more of the progress 
check test items. The segments are tabulated by performance 
both before and after remediation. 



0 
0 



0 
D 



[I TABLE 6 

Frequency Distribution of Segments 
n . . Percentage of Students 

i« Attaining At Least 80S of Objectives 

Before and After Remediation 



(f Segments =59, # Students = 44) 

Percent 

Segments 

Attaining — * 

Before Remediation After Remediat ion 
^®^ter RcT: CuirUo: No: Cum No. 



100% 3 3 35 35 

n 23 26 22 57 

0 



It'll I • 47 b 58 

60-69 6 53 1 Sfl 

50-59 1 



T"^^ » 54 

40-49 4 It 

30-39 



20-29 
10-19 
0- 9 



58 

I 59 



0 



In judging adequacy of performance, WLC maintains a 
criterion level of 80-80, that is, 80S of the students must 
achieve 801 of the objectives, if not on the first iteration 
then on the second iteration . (after remediationY or iri' 
individual tutoring sessions. The "After Remediation" 
figures in Table 6 indicate that only one segment failed 
to meet the criterion level of performance. However, in 
Table 6 "Before Remediation" data show that .20 segments did 
not attain the 80-80 criterion if segment performance is 
judged against the more stringent standard of first 
iteration performance alone. These 20 segments (and their 
progress checks) were scrutinized and revised prior to 
production of the final course materials. 

It is of interest to note that of the 50 test items 
cited in Table 5 as poor performers, 31 are found in those 
segments which failed to achieve the 80-80 criterion oh the 
first iteration. Apparently, these individual test items 
which had low scores contributed to lowering the performance 
of the entire segment. 

Tabulation of mean scores on progress checks yields 
a different analysis of progress check data which is more 
flexible for purposes of comparison than the classification 
of segments according to the percentage of students who meet 
the 80% performance criterion. 
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TABLE 7 



fl 
II 



Progress Check Mean Percentage Cot>rect 

(Before Remediation), 
Mean Time, Student Rating (High/Low) 
Of Interest Level Of Materials 



Segment Mean Mean 
Number Percentage Time 

Correct fNumber 
Minutes) 



1 1 

X • X 


o/ i 


90 


1 7 

X • 6 


HA 

OH - 




^ • X 


fifi - 
oo 


51 


c • c 


ft A 
oO 


48 




7ft 


76 


2.4 


78 
/ o 




2.5 


90 


58 


2.6 


82 


68 


2.7 


72 


.60 


2.8 


86 


60 


2.9 


90 • 


52 


3.1 


73 


50 


3.2 


80 


43 


3.3 


70 


48 


3.4 


87 


50 


3.5 


93 


47 


4.1 


92 


61 


4.2 


95 . 


52 


4.3 


84 


. 52 


4.4 


95 


67 


4.5 


95 


59 


4.6 


90 


SO 


4.7 


90 


48 


5.1 


89 


S5 


5.2 . 


76 


57 


5.3 


92 


41 


5.4 


84 


5G 


5.5 


82 


46 


5.6 


82 


40 


5.7 


83 


29 


5.8 


90 


39 


5.9 


79 


54 


5.10 


90 


34 



Percentage of Students Rating 
Interest Level of Materials as: 

High Low 



82.5 

39.4 

65.9 

46.3 

18.7 

35.7 

36.6 

53.9 

48.7 

55.5 

71.1 

13.5 

30.0 

26.3 

13.8 

61.5 

41.0 

35.9 

38.4 

32.5 

33.3 

35.9 

40.5 

25.7 

22.5 

24.4 

25.0 

19.5 

28.2 

22.5 

13.5 

10.3 

15.4 



2.5 
18.2 
2.3 
9.8 
.23.3 
11.9 
12.2 
12.9 
0.0 
2.8 
2.6 
16.2 
10.0 
15.8 
24.3 
5.1 
23.1 
20.5 
12.8 
17.5 
15.1 
15.4 
10.8 
12.9 
10.0 
9.8 
22.5 
19.5 
15.4 
20.0 
8.1 
10.3 
10.3 



TABLE 7 continued 



Segment 


Mean 


Mean 


Number 


Percentage 


Time 




Correct 


(Number 






Minutes) 


6.1 


90 


67 


6.2 


85 


63 


6.3 


88 


60 


6.4 


87 


44 


7.1 


81 


41 


7.2 . 


87 


46 


7.3 


80 


38 


7.4 


78 


4." 


7.5 


83 




8.1 


87 


64 


8. 2 


91 


66 


8.3 


92 


57 


8.4 . 


94 


44 


8.5 


87 


50 


8.6 


87 


50 


9.1 


91 


42 


9.2 


91 


42 


10.1 


95 


61 


10.2 


91 


50 


11.1 


82 


42 


♦ 11.2 


93 


26 


11.3 


90 


46 


12.1 


93 


53 


12.2 


82 


61 


12.3 


93 


36 


12.4 


83 


42 


MEAN 


86 


51 



Percentage of Students Rating 
Interest Level of Materials as 



High 


Low 


17.1 


12.2 


24.4 


14.6 


15.4 


12.8 


32.4 


5.4 


18.0 


12.9 


1 n X 


7 7 


15.4 


12.9 


13.1 


15.8 


37.5 


2.5 


29.3 


21.9 


35.0 


17.5 


29.3 


12.0 


47.5 


2.5 


33.3 


5.1 


43.6 


7.7 


22.5 


10.0 


22.5 


15.0 


17.0 


29.5 


33.3 


5.1 


19.5 


9.8 


24.4 


9.8 


38.5 


5.1 


72.1 


0.0 


50.0 


2.3 


58.5 


4.9 


43.6 


12.8 


341 


121 



Table 7 reports by segment the average percentage 
of progress check test items answered correctly before 
remediation, as well as the average time students spent 
on the instructional material in each segment and the 
interest they indicated in the materials. ' Student time 
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and attitude arc discussed in detail in the following 



Section, IV. C. The remainder of this section is a discussion * 
of mean progress check performance. 

Certain restrictions are inherent in interpreting the 
differences in progress check performance among the segments 
which are shown in Table 7. The interpretation difficulties 
spring from the fact that the segments cover different content, 
are taught by different media and forms of presentation and 
are tested by different progress check items. Despite these 
reservations it can be seen from Table 8 (derived from Table 7) 
that^ for whatever reasons, performance is considerably better 
on some segments than on others. 



TABLE 8 



Frequency Distribution of Segments by 
Mean Performance Before Remediation 



Mean 

Performanc<^ 

Progress 

Checks, 

Before 

Remediation 



Number 
Of 



Segments 



• Cumulative 
Number 



95-991 

90-94 

85-89 

80-84 

75-79 

70-74 



4 

20 
13 
14 
4 
4 



4 

24 
37 
51 
55 
59 



Table 8 reveals that even before remediation most of 
the segments are performing at a level equal to or higher 
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than the criterion level of 80%. There are 8 segments in 
this table, however, which have average scores of less than 
801, indicating that they Require revision. These segments 
have already been identified as needing revision on the 
basis of the 80-80 criterion. 

At first blush these figures may appear to contradict 
the data in Table 6 which identify 20 segments as needing 
revision because they do not meet the 80-80 criterion. This 
occurs because Table 6 records the percentage of students 
achieving 801 or better on a progress check, and does not 
reflect the distribution of scores above 801 correct. To 
illustrate: in a class of 5 students there are 4 students 
who score 100% and 1 student who scores 701. Table 6 
• would report this data as 80% of the students (4 out of 5) 
attaining 801 or better; Table 8 would report the same data 
as having a mean performance of 94%. Thus, although 
Table 6 shows that before remediation there are 20 segments 
no achieving the 80-80 criterion. Table 8 identifies only 
8 segments which do not have an average score of 80% or 
better before remediation. By averaging the performance data 
for all segments in Table 7, it is seen that the average 
percentage correct before remediation is 86%. 

Table 9 offers insight into the general improvement 
in the course effected by the revision cycle between the first 
(Spring 3970) and second (Fall I979) implementations. 



TABLE 9 



Progress Checks - First 
Mean Percentage Cor 



Segment* 


First 


Second 


Number 


Run 


Run 


1.1 


■ 

82 


87 


1.2 


67 


84 


2.1 


78 


88 


2.2 


68 


86 


2.3 


65 


70 


2.4 


71 


78 


2.5 


65 


90 . 


2.6 


71 


82 


2.7 


63 


72 


2.8 


89 


86 


2.9 


65 


90 


3.1 


66 


73 


3.2 


66 


80 


3.3 


75 


70 


3.4 


79 


87 


3.5 


84 


' 93 


4.1 


83 


92 


4.2 


85 


95 


4.3 


87 


84 


4.4 


92 


95 


4.5 


82 


95 


4.6 


73 


90 


4.7 


77 


90 


S.l 


78 


89 


5.2 


70 


76 


5.3 


74 


92 


5.4 


69 


84 


5.5 


59 


82 


5.6 


70 


82 


5.7 


80 


83 


5.8 


90 


90 


5.9 


78 


79 


5.10 


66 


90 



Second Implementations • 
, Before Remediation 



Segment 


First 


Second 


Number 


Run 


Run 


6.1 


80 


90 


6.2 


82 


85 


6.3 


91 


88 


6.4 


87 


87 


7.1 


69 


81 


7.2 


83 


87 


7.3 


77 


80 


7.4 


78 


78 


7,5 


74 


83 


8.1 


82 


87 


8.2 


81 


91 


8.3 


.92 


92 


8.4 


76 


94 


8.5 


77 


87 . 


8.6 


77 


87 


9.1 


80 


91 


9.2 


88 


91 


10.1 


93 


95 


10.2 


89 


91 


11.1 


80 


82 . 


11.2 


91 


93 


11.3 


83 


90 


12.1 


87 


93 


12.2 


85 


82 


12.3 


94 


93 


12.4 


85 


83 



MEAN - 
ALL 

SEGMENTS 7C 86 
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Folloxifing revision, performance decreased on S 
segments, remained unchanged on 3 segments, and improved 
on the remaining 51 segments. The average percentage of 
progress check scores across all segments increased by 
8% from 78% in the first run to 86S in the second, 

C' Evaluation Based on Student Attitude . 

Student attitudinal data Mere collected on a Module Question 
naire (Appendix C) which the students were aslced to complete 
at the end of each segment. It vsls felt that the questionnaire 
would be filled out with greater candor if the students felt 
sheltered by anonymity so no attempt was made to link student 
names with completed questionnaires. Most of the 44 student 
complied vith WLC's request that the questionnaires be 
completed and the number of students responding to the 
questions is at least 40 for each segment. 

Some of the items on the questionnaire were useful 

c 

primarily in revising the course materials; these items 
pertained to the difficulty of the material, the quantity of 
military examples used in the materials, the student's approval 
of the way the material was presented (media and presentation 
variables), and an open-ended question soliciting student 
opinions. The questionnaire data tabulated in. this report as 
evaluative measures come from two items - the time the students 
spent studying instructional materials and their rating on 
a 5 - point scale of the interest level of the materials (both 
content and mode of presentation) . These data are recorded 
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in Table 7 (page 30) of this report. 

Although the actual time reported for a segment varies 

considerably among students, the average time/segment as 

reported in Table 7 is the most useful measure of segment 

cluration. There is a wide spread in mean times for the 

segments, ranging from an average of 29 minutes for Segment 5.7 

to an average of 90 minutes for Segment 1.1. The average 

time across all segments, however, is 51 minutes which is 

very close to WLC's general .guideline of developing segments 

which would require 50 to 60 minutes of study time. Using 

the Module Questionnaire time data, it can be estimated that 

total instructional time for the course is 3000 minutes 

(51 min. x 59 segments) or about 50 hours. This figure 

is too low to reflect the actual time the students invested 

in the course because it does not include time spent taking 

the 59 progress checks (59 x 15 minutes = 15 hours), taking 

the cumulative posttests (14 x 40 minutes = over 9 hours), 

taking the 4 USNA tests (4 x 50 minutes = over 3 hours) and 

attending the 6 depth core meetings (6 x 50 minutes = 5 hours). 

When these time figures are added to the 50 hours of 
i - 

instructional time, the total time becomes 82 hours. In 
addition to these 82 hours, all students spent some time being 
processed as they handed in materials, had their progress 
checks graded and received the next set of materials. 
Furthermore, those students who did not score the minimum of 
80% correct on the first talcing of the progress check wore 
required to take the time to restudy pertinent material and 
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retake the progress check. When the 82 -hour figure is 
inflated to incorporate these activities, it approximates 
100 hours. Excluding holidays, a semester at the USNA 
lasts about 14 1/2 weeks; when the 100 hours are distributed 
over 14 1/2 weeks, the average amount of time the students 
spent per week is 7 hours (lOo 4- 14.5). Seven hours is 
definitely in line with the average number of hours a student 
is expected to spend on a conventional 3-credit course in " 
which he attends 3 hours of lecture and is assigned 3.-6 hours 
of homeiirork each week. 

The foregoing discussion of time data has been 
included in the section on student attitude because it is 
sometimes the case that if students feel a course is too 
time-consuming relative to their other courses, they tend 
to view it with a jaundiced eye. The time data reported by • 
the students reveal, however, that this should not be a 
problem with the WLC course. 

In the next implementation of the course the number of 
hours the student spends will decrease considerably because 
the cumulative posttests (9 hours) will not be administered, 
the materials will be distributed in larger blocks thereby 
facilitating processing, and the segments which did not 
meet the 80-80 criterion will be improved so the incidences 
of remediation and tutoring will decline. For the next 
implementation the time the student must invest in the course 
will likely not exceed 6 hours per week. 
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The second type of student attitiidinal data, reported 
as an effectiveness measure because it impacts student 
• acceptance of the course, is the students' rating of their 
interest in the materials. In response to question number 
2 on the Module Questionnaire, "Was the material interesting?/' 
the students circledthe description they felt most appropriate. 
Summary data on the responses to this question are reported 
in Table 7, page 30, with the Questionnaire ratings of "high" 
and "above average" collapsed into a single category 
representing "high" interest and the ratings of "below 
average" and "low" similarly collapsed into a single "low" 
category. These data in Table 7 are reported as the 
percentages of students who express high or low interest in 
the materials. Not reported are the percentages of students 
who rate the materials "average" - this can be calculated by 
. subtracting from 1001 the summed percentages of students in 
the high and low categories. In five segments employing 
audiotapes the percentage of students selecting the low 
category exceeds the percentage of students in the high 
category. In two other segments the high and low percehtages 
are equal. In the remaining 51 segments more students found 
the material to be of high rather than low interest. The 
averages for all segments are: 34% of the students found 
the materials to be of high interest; 121 of low interest; and 
the remaining 521 thought them to be of average interest. 



V. SEGMENTS-WITHIN-MEDIA EFPECTIVENESS 

As previously discussed, differences in segment 
performance can be attributed to a multitude of factors such 
as differences in content, test items, media and presentation 
forms and the staff who developed the materials. Thus when 
differences in segment performance arise, the difficulty 
lies in determining the causative factors. The one factor 
which may most easily be isolated is that of the medium 
employed. To obtain some estimate of the influence the 
medium itself has on segment performance, one may average 
the results for material developed in each medium and contrast 
. the averages. This has been done in Table 10. 

One important qualification must be made in interpreting 
the results in Table 10. Although the materials have been 
grouped on the basis of media, the results should not be 
construed as evidence of the superiority or inferiority of 
one medium vis a vis another. These results do not reflect 
inherent qualities of the media as such, but are rather 
indications of the effectiveness of the materials which were 
developed for and presented in each medium. The reason for 
grouping and reporting results by media is to localize the 
variations in effectiveness of materials Mch may be 
attributable to teaching via different media. The results do 
not indicate comparisons of media made over identical content 
with identical test items, developed by the same x^riter and 
employing identical presentation variables." 
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In compiling the data for Table 10 it was often 
necessary to use data reported by module rather than by 
segment because the segment contained modules developed in 
more than one medium. Progress check performance data in 
Table 10 are derived from the figures in Appendix D which 
present a breakdown of student performance on progress 
checks by module. (The research report will offer in-depth 
analyses of the presentation and media variables manipulated 
within these modules.) The time and interest data in Table 10 
are similarly tabulated on a "by module" basis. 



n 

0 
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TABLE 10 

Progress Check Mean Performance (Before Remediation)^ 
Mean Student Time, Percentage of Students Rating 
Materials as Having High/Low Interest, Averaged 
Across All Modules Developed in Each Medium 



Media 


Progress 
Check 
Perform- 
ance 1 


Time 

No. 
Mins . 


% Students Rating 
Interest Level • 

High Low 


Audiotape/ I PB 


93 


57 


36 


10 


Audiotape Script/IPB 


93 


• 54 


44 


7 


Audiotape Script/ 
Panelbook 


89 


41 


24 


11 


CAI 


88 


48 


68 


6 


Linear Text 


88 


55 


. 31 


18 


CAI Script/IPB 


87 


47 


61 


4 


Syndactic Text 


87 


52 


38 


8 


Audio tape/Panelbook 


82 


41 


23 


14 



In Table 10 the media are ranked, high to low, according 
to their progress check performance. To assess the relationsh 
between performance and the other three items Spearman rank 
order correlations between performance and time, performance 
and high interest, andperformance and low interest have been 
computed. The correlational values yielded are: 

.595 (performance -time) 

.150 (performance -high interest.) 

~.020 (performance -low interest) 

Although the correlation between performance and time is 
relatively high, with an n of 8 (8 media) the correlation is 
not significantly different from zero. Similarly, the other 
two correlational values are not statistically significant. 
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VI. REVISION PROCEDURnS FOR COURSE IMPROVEMENT 

In Section IV reference is made to the revision of 
course materials based on empirical data from the second 
implementation. The materials had already undergone one 
revision cycle before they were tested out in the second 
implementation. During the first revision cycle the extant 
materials were improved considerably (as shown in Table 9, 
page 34, average performance increased from 781 to 861), 
two media were dropped and two new ones added. 

As explicated in the Phase I Evaluation Report , the 
decision was made to omit two media - videotape/panelbooks 
and learning activity summaries - and, to replace them by 
audiotape panelbooks and syndactic texts ,. respectively . 
The media which had performed best during the first 
implementation were computer assisted instruction (CAI) and 
audiotape/intrinsically programed booklet (AT/IPB). To 
ascertain if the segments developed in these media performed 
well primarily because of the hardware they employed, it was 
decided to prepare parallel paper versions of these segments 
and to offer them as media options to the hardware-concentrated 
modules. Thus was born module 5 in Segments 4.4 - 4.7 
(AT script/IPB) and in 12.1 - 12.4 (CAI script/IPB) . Referring 
to Appendix D, the reader will see that average performance 
on module 5 did not vary much from average performance on 
the modules using audiotapes or CAI. 
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The first revision cycle therefore entailed revision 
of most segments and the development of some segments in new 
media presentations, either as substitutes for media which 
I were deleted or as additions to the current media selection, 
ji The secondrevision cycle (which followed the second 

implementation of the course) was somewhat different from 
[i the first in that: (1) revision to materials was oriented 

toward improving those specific segments and test items 
- vhich shewed low performance rather than the more global 
|j upgrading of all the course materials which took place during 

the first revision; (2) a new element was added, namely 
|[ revision of the content outline; and (3) the step toward 
JI developing a hardware-free version of all core content was 

completed by developing a syndactic text option for all 
j| audiotape/panelbook segments. These three aspects of the 

. second revision cycle are discussed in this section under two 
(J headings which separate them into revision activities based 
jl either on hard data or on subjective professional judgement. 
A. Use of Effectiveness Dat a 

Of the 59 core segments, 20 were identified (see Table 6, 
^ ■ page 28) as needing improvement on the basis that they failed 
I to meet the most stringent criterion level of performance 
I whereby 80% of the students would correctly answer 80% of the 

progress check test items on the first try. These segments 
I were examined by WLC writers and subject matter experts who 
revised material with' which the students had experienced 
difficulty. Examples used as concrete illustrations of 
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principles were sharpened, and the progress check items were 
modified by altering the stem to remove any ambiguities and/or- 
altering the responses so that the correct response more 
closely corresponded to what was taught in the materials. 
Likewise, the distractors which had draxm many responses were 
examined and modified. 

In addition to using their professional expertise to 
hone the quality of the instructional materials, the WLC 
personnel used computer printouts of progress check test item 
analyses to guide them in the revision process. The printouts 
listed the number and percentage of students who had selected 
each response to each item on the progress checks both before 
and after remediation. This data provided a clear picture of 
which objectives were misunderstood on the first attempt but 
were grasped after remediation (indicating that perhaps the 
test item or a particular aspect of teaching the objective 
needed revision), and which objectives were not attained 
either before or after remediation by many students (revealing 
most likely that instruc^.ion on the entire objective should 
be clarified or amplified; . 

Furthermore, data from the test item analyses were used 
as a basis for revising all test items which were .not correctly 
answered by 80% of the. students on the first try whether or 
not the progress check belonged to the 20 segments which 
were revised because of failure to meet the 80-80 criterion. 



Tabic S (page 27) shows that 76.21 of the test items were 
correctly answered on the first iteration of the progress 
checks. The remaining 24. 8 t were modified in light of the 
data on the test item analyses. Thus the entire revision 
process relied heavily on the empirical data from the second 
implementation of the course in preparing the materials and 
progress checks in their final form. 

Mention should be made here that the administrative 
pre/posttest was not modified before being given to the 
stidents involved in the third implementation. 

In order to revise the administrative test, in depth 
analysis comparing pre and posttest performance from the 
•fall of 1970 run was required to identify any inadequate 
test items. The course was run for the third time during 
the second semester (Spring) of 1971 and the time interval 
of one week between the administration of the posttest to 
the fall of 1970 students and the. administration of the 
pretest to the spring of 1971 students did not permit the 
requisite analysis. After -the pretest had been administered 
to the third group of students, analysis of the data from 
the second group revealed a number of test items, (see 
Table 3, page 23) on which performance was poor. Before 
administering the posttest to the third group of students, 
the above-mentioned items will be analyzed to determine 
whether they should be excluded from the posttest. 
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The empirical results of the second implementation 
j also served as the foundation for what was ultimately a 

subjective decision on the media selections to be incorporated 
I in the final production of course materials. Data from the 
first implementation had revealed that segments developed 
in vidcotape/panelbook and learning activity summary modes 
had low .overall performance; these media were dropped from 
the course since efforts to revise them did not seem warranted. 
! In a similar fashion, the media employed in the second 
j implementation, were evaluated by assessing the effectiveness 
: of the segments developed within them. Table 10, page '40 
^ lists the average effectiveness of segments developed in each 
of the media - the lowest average performance occurs in the 
11 audiotape/panelbook segments where the mean performance 
j^j before remediation is 821 which, although it is the lowest 
score, still exceeds the 80% which WLC set as the minimum 
level of acceptable performance. 
J The 20 segments which did not meet the 80%-80% criterion 

on the first iteration (Table 6, page 28 ) are grouped by 
media in Table 11 to yield a clearer picture of possible media 
influence on poor performance on some segments. 
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Table 11 



Identification by Media of the 20 Segments 
• Wot Meeting the 805/80% Criterion, 
First Iteration (Reference Table 6, page 28) 



Audiotape/ 

Panclbook 

Segments 


Linear 
Text 

Segments 


Syndactic 
Text 

Segments 


2.3 5.9 

2.4 7.1 

3.1 7.3 

3.2 7.4 

3.3 7.5 


5.4 
5.5 
5.6 


2.6 
2.7 
5.2 
8.6 
11.1 



CAI 

Segments 

12.2 
12.4 



Because half of the 20 segments which performed poorly 
.were developed in the medium of audiotape/panelbook (AT/PB) , 
the possibility of eliminating this medium altogether was 
considered. However, since overall performance on this 
medium was at an acceptable level (821), the decision was ' 
made to retain it and to give attention to optimizing the 
effectiveness of the medium per ste in all AT/PB segments as 
well as to subjecting the specific 10 segments to the 
standard revision process. Since instruction in the AT/PB 
medium is so heavily oriented toward aural delivery, 
.attention was given to stylistic and delivery variables as 
well as to content itself. WLC recognized the basic 
differences between written and auditory presentations and 
took pains to revise the impact of the auditory style of 
instruction. This entailed rephrasing of the audiotape 
scripts and retaping each of the 19 AT/PB segments. The 
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panelbooks which were used as an adjunct to the taped lectures 
were le£t virtually unchanged. 

« 

B. Subjective Evaluation . 

WLC concurred with the USNA's suggestion to render 
implementation of the course more flexible by delivering a 
paper version to parallel every core segment tied to 
hardware. The syndactic text mode of presentation was 
selected to carry the paper version of the AT/PB segments 
because syndactic text is essentially a linear programed 
text which incorporati^s an additional element of management 
in the guise of the summary and summary quiz. A major 
•effort, therefore, in the second revision cycle was tho 
generation of parallel syndactic texts for the 19 AT/PB 
segments. 

In "syndacticizing" these segments, the writer divided 
an audiotape script into approximately three major content 
areas and wrote a summary for each area. A summary quiz was 
then developed to measure comprehension of the content 
presented in the summary. The next step was the generation 
of linear frames to be used as remediation by the student 
who did not correctly answer all questions on the summary 
quiz. The linear frames taught essentially the same content 
as. the summary but provided instruction in smaller steps 
and frequently supplemented the instruction with examples. 
Portions of the panelbook, such as diagrams and charts, were 
included in the syndactic text where they were essential to 
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instruction. Many pictures and illustrations in the 
panelbopk were eliminated from the syndactic text when 
their presence was not requisite to the learning of 
the material . 

Subjective evaluation of content, with regard to 
appropriateness and level of specificity became a major 
area of concern during this final revision. It was the 
point of view of the contractor that the content outline 
had been developed along a spiral or helical structure. 
This pattern of organization suggests that compliex notions 
are i 3st taught in recurring cycles which allow the 
learner to look at many facets of the same concept. 
Specifically, instructional material dealing with communication 
should not be relegated to that section entitled Interpersonal 
Communication:, but should be reintroduced as it relates to 
other areas of content such as Morale, or Senior Subordinate 
Relations. According to the logic of the spiral construct 
a chart or graph which serves a specific purpose in one 
chapter can be purposely introduced again in another section 
to focus on another aspect of the issue represented. It 
was unfortunate that this approach was deemed by the customer 
to cause redundancy. indeed, the question of redundancy 
and necessary changes to the content out j ine was a clear 
conflict of views. The data compiled by the contractor 
indicated not only a high student interest level in the 
materials, but additionally, in response to question 7 
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on the Module Questionnaire asking, "Did you feel that 
there were any sections of the materials that were 
unduly repetitious?," a mean of 93.81 of all students, 
answering this question replied in the negative. Therefore, 
it was with a great deal of reluctance that the contractor 
shared the. efforts in deleting previously agreed upon 
materials from the content outline. 

The scope of this effort entailed the re -examination 
of the skeletal framework of each segment, and the actual 
revision of 80% of the segments (32 segments were 
extensively revised and 14 had minor changes) . These 
changes to the content outline necessitated corresponding 
alterations to materials and test items. 
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COURSE OUTLINE 



PART ONE; Ovorviev/ of Leadership 

Segment I Concepfs of Leadership ^ 

Segment II Standards of Leadership in the Novel Service 

PART TWO: Individual Behavior 



Segment I 
Segment 11 
Segment ill 
Segment IV 
Segment- V 
Segment VI 
Segment VH 
Segment VIII 
Segment IX 



Introduction to Psychology 
Behavior and Its Observation 
Learning 

Factors Affecting Learning 
Attention, and Perception 
Motivation 
Conflict 

Neurotic and Psychotic Reactions 
Personality 



PART THREE; Group Dynamics 

Segment I Characteristics of Groups 

Segment II The Relation of the Leader to the Group 

Segment III Group Interactions 

Segment IV Conformity as a Factor of Group Behavior 

Segment V Relation of the Individual to the Group 

PART FOUR; Achieving Effective Communication 

Segment I Importance of Interpersonal Communication 

Segment II Types of Communication 

Segment III . The Communication Process (Receiver and Barriers) 

Segment IV The Communication Process (Sender and Feedback) 

Segment V Formal Communication and Its Dimensions 

Segment VI Informal Communication 

Segment VII Communication Under Battle Situations 

PART FIVE: Military Management 



Segment I 
Segment II 
Segment III 
Segment IV 
Segment V 
Segment VI 
Segment VII 
Segment VIII 
Segment IX 
Segment X 



Introduction to Management and the Management Process 

Decision Making and Creativity 

Objectives • . 

planning 

Organizing: Principles and Process 

Organ i 2 i ng^ Struc ture 

Organizing; Charting 

Directing 

Controlling 

Coordinating 



APPENDIX A 
Continued 



The Leader 

The Group and The SItuaHon 



PART SIX; Authority ond Responsibility 

Segment I Concept of Authority 

Segment 1 1 Why People Accept/Resist Authority 

Segment III Delegation of Authority; Line-Staff Relationship 

Segmeht IV Responsibility 

PART SEV EN; Leadership Beha vior and Style 

Segment I Leadership Behavior 

Segment II Leadership Style 

Segment III Determiners of Leadership Style 

Segment IV Determiners of Leadership Style 

Segment V Participative Leadership 

PART EIGHT; Senior-Subordinate Relationships •: ' ' 

Segment I Organizational Structure-& Social Distance in Senior- • ' " 

Subordinate Relationships 

Segment II Officer-Enlisted Relationships 

Segment III Assumption of Command and Formal & Informal Leader Relationsh 

Segment IV Introduction to Counseling 

Segment V The Counseling Process 

Segment VI Relations v/ith Seniors and Contemporaries 

PART NINE; Morale - Esprit de Corps 

Segment I Morale 

Segment II Group Solidarity and Esprit 

PART TEN; Discipline 

Segment I Introduction to Discipline 

.Segment II Development ond Maintenance of Discipline 

PART ELEVEN; Personnel Evaluation 

Segment I The Role of Evaluation 

Segment II Enlisted Performance Evaluation 

Segment III Officer Evaluation 

PART TWELVE; " Applied' Leodership 

Segment I . Measvrement of Effective Leadership 

Segment 11 Generally Recognized Characteristics of an Effective Leader 

Segment III Techniques of Assuming Command 

Segment IV "That's an Order I" 
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PRESENTATION DESIGN 



FALL OF 1970 IMPLEMENTATION 
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APPENDIX C 



For the following questions cireic the appropriate munbor. 



I 



Was the material inter cstlns:? 



as the material difficult? 



j ale your approval of the way tiio 
l-.alerial was presenlecl (i.e. 
syndaclic text, audiotape, ote.) 

[•jany points were illustrated by 
mi litary exampl es . Rate your 

5sessment of their instruetlonal 
wfeeti veness. 



I^ow did you fee) about (he 
pantiJy of military examples usecf? 



1 

1 



Above 
2 

« 

2 



3 



Below 

4 
4 



5 
6 



Too 
Many 



About 



Too 
Few 



2 



[ |-jicl you feel that there were any seetlons of the materials that were vmduly repetitious ? 
f If you answered yes to the above question, please specify the seetIon(s) by page mm'ber. 



you have any additional statements to make about any of the above questions, please write 
*.ie question number and make yoin: eommcnt here. 



Hiat suggestions would ycu have for improving this segment? 



! 
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APPENDIX D 

Mean Progress Check Performance 
(Before Remediation) 
By Module Studied 



Audiotape/Intrinsically Programed Booklet 
Audiotape Script/Intrinsically Programed Booklet 



ll 

ill 
I li 

t ■ 

lit 

i 

ill. 

0 

[l] 

0. 

-"To . 



Segment 
Number 



4.4 
4.5 
4.6 
4.7 

Mean 



Total 
Segment 
Perform- 
ance 

95 
95 
90 
90 

93 



Mod. 
Tape 
HRDF 

mv 


1 Mod. 2 
Tape 
HRDF 
MMF 


Mod. 3 
Tape 
HRDF 
LMF 


Mod. 4 
Tape 
LRDF 
LMF 


Mod. 5 
Script 
HRDF 
HMF 


96 


94 


98 


96 


91 


97 


97 


90 


94 


96 


90 


90 


92 


84 


91 


92 


90 


91 


90 


89 


94 


93 


93 


91 


92 



Mean for all tape modules «= 93 
Mean for all script modules = 92 



Computer Assisted Instruction (CAI) 
CAI Script/Intrinsically Programed Booklet 



Segment 
Number 


Total 
Segment 
Perform 
ance 


Mod. 
CAI 
- HRDF 
HMF 


1 Mod. 

HRDF 
MMF 


2, Mod. 3 
CAI 
HRDF 
LMF 


Mod. 4 
CAI 
LRDF 
LMF 


Mod. 5 
Script/IPB 
HRDF 
HMF 


12.1 


93 


93 


90 


95 


93 


96 


12.2 


82 


81 


83 


81 


85 . 


78 


12.3 


93 


97 


91 


89 


09 


92 


12.4 - 


83 


87 


81 


78 


8S 


82 


Mean 


88 


90 


86 


86 


90 


87 


Mean for 


all CAI 


modules 


= 88 








Mean for 


all CAI 


script modules 


= 87 
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Audiatape/Intrinsically Programed Booklet 
Audiotape Script/Intrinsically Programed Booklet 



Segment Total Mod. 1 Mod. 2 Mod. 3 Mod. 

Number Segment Script Tape Script Tape 

Performance Content Content No rnnt«»«*- m« r 



10.1 95 

10.2 91 



Mod. 1 


Mod. 2 


Script 


Tape 


Content 


Content 


Map 


Map 


93 


93 


90 


93 


92 


93 



No Content No Content 

Map Map 

99 96 

88 95 



Mean 93 

Mean for all tape modules «= 95 
Mean for all script modules = 93 
Mean for all modules with content maps « 93 
Mean for all modules without content maps « 95 



94 96 



Audiotape/Panelbook 
Audiotape Script/Panelbook 



Segment 
Number 



7.S 
9.1 
9.2 

Mean 

Mean for 

Mean for 

Mean for 

Mean for 



Total 

Segment 

Performance 



83 
91 
91 



Mod. 1 
Tape 
Content 
Map 



80 
93 

92 



Mod. 2 Mod. 3 

Tape Script 

No Content 

Content Map 
Map 



84 
91 
92 

89 



85 
92 
92 



88 88 

all tape modules «= 89 

all script modules = 89 

all modules with content maps = 89 

all modules without content maps = 



90 



Mod. 4 
Script 
No 

Content 
Map 

83 
8^ 
88 

87 



88 



f 
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Audiotape/Panelbook 



Seg. 
No. 



2.2 

2.3 

2.4 

2.5 

3.1 

3.2 

3.3 

3.4 

5.7 

5.8 

5.9 

5.10 

7.1 

7.2 

7.5 

7.4 

Mean 
Mean 
Mean 
Mean 
Mean 
Mean 
Mean 



Total 
Seg. 
Perf . 



86 

70 

78 

90 

73 

80 

70 

87 

83 

90 

79 

90 

81 

87' 

80 

78 

81 

for all 
for all 
for all 
for all 
for all 
for all 



Mod.l Mod. 2 Mod. 3 Mod. 4 Mod.S 

HRDF LRDF LRDF HRDF HRDF 

Conf. No Conf. No Conf. 

Overt Conf. Overt Conf. Covert 



93 
73 
78 
88 
75 
77 
72 
85 



Overt 

85 
69 
77 
88 
73 
84 
73 
88 



Overt 



80 



80 



HRDF modules =82 

LRDF modules = 81 

modules having confirmation » 82 

modules having no confirmation = 82 

modules with overt response = 82 

modules with covert response « 82 



Mod. 6 Mod. 7' 
LRDF LRDF 
No Conf. 
Conf. Covert 
Covert 

76 
65 
78 
93 
75 
77 
69 
88 



84- 


80 


. 85 


85 


93 


88 


93 


88 


71 


87 


82 


75 


90 


88 


91 


92 


84 


83 


79 


78 


86 


86 


92 


85 


79 


79 


85 


79 


73 


80 


81 


80 


83 


84 


86 


83 



. 78 



Mod . 8 

HRDF 
No 

Conf. 
Covert 

85 
72 
80 
91 
69 
79 
68 
88 



79 



APPENDIX D 
Continued 



Syndactic Text 



Segment Total Mod. .1 Mod. 2 Mod 3 

Number Segment Remed. Remed. Remed. 

Performance Program Detailed Summary 

Sequence Summary 



2.6 


82 


78 


80 


85 


2.7 


72 


76 


68 


74 


2.8 


86 


86 


85 


88 


6.1 


90 


93 


88 


90 


6.2 


85 


89 


86 


80 


6.3 


88 


88 


88 


90 


11.1 


82 


85 


81 


80 


11.2 


93 


95 


93 


90 


11.3 


90 


95 


88 


87 


Mean 


85 


87 


84 


85 



Syndactic Text (Remed. - Programed Sequence) 



Segment 
Number 


Total 

Segment 

Performance 


Mod. 1 
Map 


1~ ' 

Mod. 2 
No Map 


5.1 


89 


89 


89 


5.2 


76 


77 


75 


5.3 


92 


93 


90 


8.4 


94 


93 


95 


8.5 


87 


86 


88 


8.6 


87 


88 


85 


Mean 


88 


88 


87 



UH 



'I 
il 



APPENDIX D 
Continued 



111 

If! 



I 

I. I 



Syndactic Texts (Remed. - Programed Sequence) 



Segment 
Number 



2.1 
2.9 
3.5 
6.4 

Mean 

Mean for 

Mean for 

Mean for 

Mean for 



Total 
Segment 
Performance 

88 
90 
93 
87 

90 

all modules with RAF's = 90 
all modules without RAF's = 90 
all modules with content maps = 91 
all modules without content maps = 



Mod. 1 

RAF 
Map 


Mod. 2 
RAF 

No Map 


Mod. 3 

No RAF 
Map 


Mod. 4 

No RAF 
No Map 


88 


87 


8.8 


86 


90 


86 


92 


90 


94 


93 


95 


92 


90 


87 


85 


88 


91 


88 


90 


89 



89 



Map=Content Map 

RAF=Revealed 
Answer Form 



Linear Texts 



ERIC 



Segment 
Number 



4.1 
4.2 
4.3 
5.4 
. 5.5 
5.6 
8.1 
8.2 
8.3 
Mean 

Mean for 
Mean for 
Mean for 
Mean for 



Total Mod. 1 
Segment HRBF 
Perform- Overt 
Under- 
line 



Mod. 2 Mod. 3 Mod. 4 

HRDF HRDF LRDF 

Overt Covert Overt 

Selected Under- 
line 

95 91 84 . 98 
99 94 96 93 

85 84 87 " 85 
83 84 86 77 
79 89 80 78 

86 84 83 78 
89 . 83 90 87 
91 91 93 91 

96 . 96 85 91 
89 88 87 86 

all HRDF modules =88 

all LRDF modules - 87 

all overt-underline modules = 88 

all overt-selected modules = 88 



ance 

92 
95 
84 
84 
82 
82 
87 
91 
92 
88 



Mod. 5 Mod. 6 

LRDF LRDF 
Overt Covert 
Selected 



93 
94 
83 
87 
80 
80 
86 
90 
94 
87 



93 
94 
83 
85 
84 
83 
88 
93 
88 
88 



